docs(factories): document measurement and improvement by hongyi-chen · Pull Request #522 · warpdotdev/docs

hongyi-chen · 2026-08-13T19:17:28Z

Summary

Organizes dashboard metrics, scorers, benchmarks, and autofix around the questions they answer and their limitations. Manual/periodic scoring and a six-step evidence-driven improvement loop form the operational path.

Final size: 883 prose words. Across the section, the senior editorial pass reduced prose from about 14,600 to 7,649 words while preserving verified behavior and security caveats.

Foundation

Shared navigation, route placeholders, Early Access badge support, and guide migrations are merged in #537. This PR now contains only its feature-owned files and passes CI independently.

Validation

Integrated npm run typecheck: passed
Integrated npm run build: 377 pages built successfully
Integrated internal-link check: 3,566 links checked, 0 broken
Page-specific style lint: 0 errors (glossary-candidate warnings only where noted)
Senior editorial, product-accuracy, and security/public-safety reviews completed; all blocker and important findings resolved

Latest source refresh

Adds current Dashboard metrics, Scorer terminology, Self-improvement surfaces, pr_facts limitations, Time saved/Autonomy/latency/run breakdown, and redesigned Benchmarks.

Verified against Warp e72fd7aac and warp-server 9be39e484b. Broken, placeholder, partial, and spec-only surfaces remain excluded.

Proposed reviewers

Based on the Warp Factories Soft Launch (August 18th) tracker. For planning only; no review requests have been sent.

@szgupta
@cephalonaut
@coolcom200

Screenshots

Not included. The revision uses sourced tables, Mermaid diagrams, and verified code/config examples; no safe approved Factory UI assets exist yet.

Unverified claims

None — all UI labels, defaults, eligibility claims, diagrams, and configuration details were verified against source or deliberately omitted.

vercel · 2026-08-13T19:17:35Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview	Aug 16, 2026 11:09pm

oz-for-oss · 2026-08-13T19:30:32Z

@hongyi-chen

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR adds the Factories measurement and improvement documentation, including metrics, scorers, benchmark suites, autofix, and an improvement-loop workflow. The content is generally aligned with the docs style guide, and I did not find security concerns or spec-drift issues from the provided context.

Concerns

The new page links readers to Factories pages that are not present in the attached diff, which risks publishing broken internal links.

Verdict

Found: 1 critical, 0 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss · 2026-08-13T19:34:11Z

+* [**Factory agents**](./factory-agents) - Choose responsibilities, models, harnesses, skills, and tools for specialized roles.
+* [**Factory definitions as code**](./factory-as-code) - Review changes to agent prompts, skills, and factory configuration.


🚨 [CRITICAL] These related-page links point to ./factory-agents and ./factory-as-code, but neither target page appears in this diff; add those pages before publishing or link to existing Factories pages so the docs link check does not fail.

hongyi-chen · 2026-08-16T00:01:46Z

/oz-review

oz-for-oss · 2026-08-16T00:01:59Z

@hongyi-chen

I'm re-reviewing this pull request in response to a review request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR replaces the placeholder Factories measurement page with a complete guide to dashboard metrics, Scorers, benchmarks, Self-improvement, and a practical evidence-driven improvement loop.

Concerns

No blocking concerns found in the attached diff.

Verdict

Found: 0 critical, 0 important, 0 suggestions

Approve

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

Co-Authored-By: Warp Agent <agent@warp.dev>

hongyi-chen · 2026-08-16T01:35:18Z

/oz-review

oz-for-oss · 2026-08-16T01:35:29Z

@hongyi-chen

I'm re-reviewing this pull request in response to a review request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss

Overview

This PR replaces the factories measurement placeholder with a full page covering dashboard metrics, Scorers, benchmarks, Self-improvement, and a practical improvement loop. The content is scoped to the changed page, and the security/spec passes did not identify material issues.

Concerns

Two wording issues make otherwise useful explanations harder to parse; both have inline suggestions.

Verdict

Found: 0 critical, 0 important, 2 suggestions

Approve with nits

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

oz-for-oss · 2026-08-16T01:38:40Z

+
+## Configure Scorers
+
+A user-defined **Scorer** tells an LLM judge how to classify completed conversations. User-defined Scorers have no built-in rubrics and currently support classification rather than numeric quality evaluation. They are separate from built-in **Correctness**, a Warp-managed scorer that benchmarks run automatically. Define one user-defined Scorer for one decision so its failures remain actionable.


💡 [SUGGESTION] Rewrite this clause so it says Correctness runs automatically in benchmarks, rather than that it "benchmarks run automatically."

Suggested change

A user-defined **Scorer** tells an LLM judge how to classify completed conversations. User-defined Scorers have no built-in rubrics and currently support classification rather than numeric quality evaluation. They are separate from built-in **Correctness**, a Warp-managed scorer that benchmarks run automatically. Define one user-defined Scorer for one decision so its failures remain actionable.

A user-defined **Scorer** tells an LLM judge how to classify completed conversations. User-defined Scorers have no built-in rubrics and currently support classification rather than numeric quality evaluation. They are separate from built-in **Correctness**, a Warp-managed scorer that runs automatically in benchmarks. Define one user-defined Scorer for one decision so its failures remain actionable.

oz-for-oss · 2026-08-16T01:38:41Z

+
+## Compare configurations with benchmarks
+
+A benchmark suite fixes one agent and compares configurations of it on the same tasks. Define the suite, then launch it with the configurations you want to test:


💡 [SUGGESTION] Use the possessive form to make this sentence easier to parse.

Suggested change

A benchmark suite fixes one agent and compares configurations of it on the same tasks. Define the suite, then launch it with the configurations you want to test:

A benchmark suite fixes one agent and compares its configurations on the same tasks. Define the suite, then launch it with the configurations you want to test:

Rewrite the page in plainer language based on review feedback: - Rename the jargon table headers (Boundary/Limitation -> Keep in mind) - Drop the 25-lines-per-engineer-hour heuristic explanation; describe Time saved simply as a directional estimate - Merge the two dense metric-caveat paragraphs into one - Shorten Scorer, benchmark, Self-improvement, and improvement-loop sections without changing verified behavior claims Co-Authored-By: Warp <agent@warp.dev>

hongyi-chen · 2026-08-16T05:02:41Z

Simplified the page based on feedback that the language was too technical (b41d6ce):

Renamed the Boundary/Limitation table columns to Keep in mind and rewrote the entries in plain terms.
Cut the "25 finished lines per engineer-hour" heuristic explanation. Time saved is now described as a rough, directional estimate.
Merged the two dense caveat paragraphs after the metrics table into one, and dropped the human-touch-indicator hedge.
Shortened the Scorer, benchmark, Self-improvement, and improvement-loop sections; no verified behavior claims changed.

Validation: npm run build passes (377 pages); style lint shows only the pre-existing glossary-candidate warnings.

Reviewer feedback: docs should read as objective statements, not advice. Both tables are now two columns, with measurement facts folded into the metric descriptions and coverage facts kept in the caution block. Co-Authored-By: Warp <agent@warp.dev>

hongyi-chen · 2026-08-16T05:34:02Z

Follow-up to the simplification pass (ce81d9b): removed the "Keep in mind" columns entirely. Both tables are now two columns — measurement facts moved into the metric descriptions (e.g., "collected through the GitHub App", "based on recorded PR signals"), and cross-cutting coverage facts stay in the caution block. Build passes (377 pages); lint unchanged.

- Metric rows describe only what each metric shows; GitHub App coverage facts live in the caution block alone - Cut UI inventory (median/mean/By complexity/By size views, scoreboard, scatter plot) and billing-rate conversion detail - Replace jargon: eligible, lower-bound, freeze, clusters, PR signals, webhook data - Dedupe Self-improvement paragraphs and the three-newest-PRs fact, fix the ambiguous pronoun, drop the closing aphorism and generic benchmark advice already covered by loop step 4 Co-Authored-By: Warp <agent@warp.dev>

hongyi-chen · 2026-08-16T05:42:02Z

Final polish pass (f7e8925): removed remaining implementation details and duplication.

Metric rows now state only what each metric shows; GitHub App coverage facts appear once, in the caution block.
Cut UI inventory (median/mean, By complexity/By size views, "scoreboard", "scatter plot") and the billing-rate conversion detail.
Replaced jargon: "eligible", "lower-bound", "freeze", "clusters", "PR signals", "webhook data".
Deduplicated the Self-improvement section, removed the three-newest-PRs fact from prose (kept in the table), and dropped the closing aphorism plus benchmark advice already covered by loop step 4.

Page is now ~640 words (from 883 at review start). Build passes (377 pages); lint unchanged.

- Trim the dashboard caution to the actionable GitHub App coverage caveat; move the run-count composition into the interpretation paragraph and drop the opaque merge-rate sentence - Add a Mermaid diagram to the practical improvement loop showing the monitoring cycle and where Self-improvement joins it - Link definitions as code from the Self-improvement section, explaining that follow-up runs improve the versioned factory definition through reviewable pull requests Co-Authored-By: Warp <agent@warp.dev>

cla-bot Bot added the cla-signed label Aug 13, 2026

vercel Bot deployed to Preview August 13, 2026 19:19 View deployment

hongyi-chen marked this pull request as ready for review August 13, 2026 19:30

oz-for-oss Bot reviewed Aug 13, 2026

View reviewed changes

hongyi-chen force-pushed the hyc/factories-measure branch from 3acee02 to 020e7cc Compare August 13, 2026 20:31

vercel Bot deployed to Preview August 13, 2026 20:34 View deployment

hongyi-chen force-pushed the hyc/factories-measure branch from 020e7cc to 848d581 Compare August 14, 2026 17:43

vercel Bot deployed to Preview August 14, 2026 17:46 View deployment

hongyi-chen mentioned this pull request Aug 14, 2026

[integration check — do not merge] Factories launch docs, all pages combined #533

Closed

hongyi-chen force-pushed the hyc/factories-measure branch from 848d581 to 9e6562a Compare August 15, 2026 23:45

vercel Bot deployed to Preview August 15, 2026 23:48 View deployment

oz-for-oss Bot reviewed Aug 16, 2026

View reviewed changes

docs(factories): document measurement and improvement

aaecb28

Co-Authored-By: Warp Agent <agent@warp.dev>

hongyi-chen force-pushed the hyc/factories-measure branch from 9e6562a to aaecb28 Compare August 16, 2026 01:30

vercel Bot deployed to Preview August 16, 2026 01:33 View deployment

oz-for-oss Bot reviewed Aug 16, 2026

View reviewed changes

vercel Bot deployed to Preview August 16, 2026 05:05 View deployment

vercel Bot deployed to Preview August 16, 2026 05:36 View deployment

vercel Bot deployed to Preview August 16, 2026 05:44 View deployment

hongyi-chen requested a review from szgupta August 16, 2026 23:08

vercel Bot deployed to Preview August 16, 2026 23:09 View deployment

		* [Factory agents](./factory-agents) - Choose responsibilities, models, harnesses, skills, and tools for specialized roles.
		* [Factory definitions as code](./factory-as-code) - Review changes to agent prompts, skills, and factory configuration.


		## Configure Scorers

		A user-defined Scorer tells an LLM judge how to classify completed conversations. User-defined Scorers have no built-in rubrics and currently support classification rather than numeric quality evaluation. They are separate from built-in Correctness, a Warp-managed scorer that benchmarks run automatically. Define one user-defined Scorer for one decision so its failures remain actionable.


		## Compare configurations with benchmarks

		A benchmark suite fixes one agent and compares configurations of it on the same tasks. Define the suite, then launch it with the configurations you want to test:

Uh oh!

Conversation

hongyi-chen commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Foundation

Validation

Latest source refresh

Proposed reviewers

Screenshots

Unverified claims

Uh oh!

vercel Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Verdict

Uh oh!

oz-for-oss Bot Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

hongyi-chen commented Aug 16, 2026

Uh oh!

oz-for-oss Bot commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Verdict

Uh oh!

hongyi-chen commented Aug 16, 2026

Uh oh!

oz-for-oss Bot commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oz-for-oss Bot left a comment

Choose a reason for hiding this comment

Overview

Concerns

Verdict

Uh oh!

oz-for-oss Bot Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

oz-for-oss Bot Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

hongyi-chen commented Aug 16, 2026

Uh oh!

hongyi-chen commented Aug 16, 2026

Uh oh!

hongyi-chen commented Aug 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

hongyi-chen commented Aug 13, 2026 •

edited

Loading

vercel Bot commented Aug 13, 2026 •

edited

Loading

oz-for-oss Bot commented Aug 13, 2026 •

edited

Loading

oz-for-oss Bot commented Aug 16, 2026 •

edited

Loading

oz-for-oss Bot commented Aug 16, 2026 •

edited

Loading